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Abstract 


How do users of music information retrieval (MIR) 
systems express their needs? Using a Wizard of Oz 
approach to system evaluation, combined with a 
grounded theory analysis of 502 real-world music 
queries posted to Google Answers, this paper 
addresses this pivotal question. 


1 Introduction 


In this paper we study the issue of what facilities people really 
want when performing a music information retrieval (MIR) 
task by analysing the questions and answers posted to the 
music category of Google’s “ask an expert” service, Google 
Answers. This work builds on a previous study that analysed 
postings to a Usenet newsgroup for a specific genre of music: 
‘old time’ country (Downie and Cunningham, 2002). The set 
of music queries analysed in this present work is considerably 
larger (approximately four fold), and the scope of questions 
more broadly based. 


Google Answers is essentially a reference service affiliated 
with, but not restricted to, the Web; the music category can be 
broadly seen as an MIR system in which the ‘interface’ 
involves submission of natural language queries to be 
‘processed’ by human experts. The queries posed to Google 
Answers express authentic music information needs, not 
constrained by necessity to use an artificial query language. 
We argue this adds up to a “Wizard of Oz” system for music 
information retrieval—that is, the Google Answers experts 
simulate the processing of an MIR system. The Wizard of Oz 
technique is commonly used in computer application design to 
evaluate an approach before it is implemented, in an attempt 
to gain insights that can help evolve the design before 
irrecoverable coding decisions are made (see, for example, an 
overview in Dahlback et al, 1993). We use the approach here 
to see what sort of capabilities an MIR needs to serve a 
diverse group of users. 
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2 Data collection 


Google Answers makes an interesting case study since it has a 
sizable number of users and experts, has been running a 
reasonable length of time (over | year) and music questions in 
particular seem a popular preoccupation: music is large sub- 
category within Arts and Entertainment. 626 postings were 
retrieved from the music sub-category, spanning 10 April 
2002-1 April 2003. Note that it is the poster who determines 
the subject category associated with the question, so it is the 
users themselves who have decided that these are fitting music 
queries. However, 5 postings in our analysis were discarded as 
being off-topic (for example, “How do you find investors?”). 
An additional 119 postings were related to the music industry 
rather than to music itself, and so these postings are not further 
analyzed in this paper. 


The 502 remaining postings in the music category were 
analysed using a grounded theory approach (Glaser and 
Strauss, 1967). With this technique researchers attempt to 
approach the data without prior assumptions, and to generate 
theory from the data. The queries were coded to characterize 
the types of details users are able to offer when describing 
their information need. 


3. How do people describe what they want? 


Unfortunately, space limitations preclude a _ detailed 
explication of the analytic categories emerging from the 
queries. Table 1 shows that for the majority of music queries 
analysed (slightly over 80%), users are able to provide some 
form of Bibliographic metadata when describing their 
information request. By far the most common bibliographic 
attributes (Table 2) supplied are the name of the Performer(s) 
and the Title of a work—certainly an MIR system should 
include these metadata, as a bare minimum. A_ small 
proportion of queries included a URL Link to a webpage 
providing additional bibliographic metadata. Orchestration 
might include instrument, vocal range of singers, or (most 
commonly) the gender of the singers. The default Language 
for lyrics appears to be English, and language was only 
mentioned in the query if the desired work(s) had non-English 
lyrics. Similarly, the Nationality of the performer (Table 2) 
was primarily mentioned when that person was not from the 


Category Description of information need includes... Count % 
BIBLIOGRAPHIC metadata (see Table 2 for breakdown) 408 81.3 
GENRE Description of the genre or style 164 32.7 
LYRIC FRAGMENT some or (rarely) all of “the words” 145 28.9 
WHERE HEARD circumstances surrounding a remembered performance or broadcast of a song 121 24.1 
NATIONALITY Nationality of performer or origin of song 63 12.5 
SIMILAR reference(s) to known work(s) or performer(s) used to defined attributes upon which 

describe the unknown item(s); 23 4.6 
EXAMPLE representation(s) of the desired work(s) (e.g., links to MP3 or midi files) 22 44 
LYRIC STORY storyline of song 13, 2.6 
AFFECT ‘mood’ or emotional state induced (e.g., “funny”, “silly” , “plaintive’’) 12 2.4 
TEMPO speed and/or rhythm of work(s) 12. 2.4 
Note #1: % calculated against the set of 502 queries seeking music work(s) or music work information 

Table 1. Categories of need description types 
Category Description Count % 

Performer performer or group who created a particular recording 240 58.8 
Title Name (or approximation) of work(s) 176 43.1 
Date date that a recording was produced, or that a song was composed 160 39.2 
Orchestration Name of instrument(s) and/or vocal range(s) and/or genders (male/female) 68 16.7 
Collection title Name of album, LP, CD, audiotape, etc 61 15.0 
Composer Name of composer 36 8.8 
Label Name of organization which produced recording(s) 27 6.6 
Link URL providing a link to further bibliographic data 12 2.9 
Language specifies a particular language (other than English) for lyrics 10 2.5 
Other bibliographic data falling outside the above categories 36 8.8 


Note #1: % calculated against the set of 408Bibliographic queries 


Table 2. Breakdown of BIBLIOGRAPHIC category from Table 1. 


US or the UK. From these categories, it appears that quality 
bibliographic metadata may be crucial for use in searching. 


Users experienced difficulty in coming up with crisp 
descriptions for several of the categories, indicating a need to 
support ‘fuzzy’ or imprecise metadata values for searches. For 
example, the Date of composition or recording for the desired 
musical items is rarely specified precisely by year. More 
typically, the decade will be given, or other somewhat 
nebulous date indications such as “recent” or “old”. The user 
may be uncertain as to the accuracy of the lyrics that they 
recall (“the chorus went something like...”). Some Lyric 
elements may not be dictionary words (“bling bling”), which 
poses problems for the user in trying to come up with the 
‘correct’ spelling; it is also not clear how to transcribe 
repetitions (“Money money money moooooooneeceeeyy..... 
MO-NEY?”). Sometimes users were uncertain as to the 
accuracy of their descriptions: “female singer, African- 
American likely (but i could be wrong..heck it might just be a 
male with a female-sounding voice)”. 


Genre descriptions ranged from standard (but not crisply 
defined) characterizations (“jazz’, “pop”) to the highly 
idiosyncratic (“Sort of that teenie bop bitter angst genre”). The 
lack of consensus over genre categories and descriptions 
suggests that it may be appropriate to support (or replace) a 


genre label with examples of music falling within the genre; 
perhaps a more productive way of locating new pieces within 
a genre is to allow the user to ask for ‘more things like this’ by 
providing music query-by-example facilities 


Few queries included an audio Example, yet it would be 
premature to conclude that query-by-humming MIR systems 
have no potential user-base. Some queries suggest that users 
would like to include a musical representation but are stymied 
by the form-based interface to Google Answers: “The only 
other information that I have is what the song sounds like, but 
it is hard to express that via the web.” 
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